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ABSTRACT 



We report the result of a search for D D mixing in the data from hadropro- 
duction experiment E791 at Fermilab. We use the charge of the pion from the 
strong decay D* + — > D°tt + (and charge conjugate) to identify the charm 
quantum number of the neutral D at production, and the charge of the 
lepton and the kaon in the semileptonic decays D° — » Kev and K[iv to 
identify the charm at the time of decay. No evidence of mixing is seen. 
We set a 90% confidence level upper limit on mixing of r < 0.50%, where 
r = T(D° ^D ^ K + l-ui)/T{D° -> K~l + vi). 

(Submitted to Physical Review Letters) 

The predicted rate of D°D mixing in the Standard Model [1] is several 
orders of magnitude below the sensitivity of current experiments. However, 
several theoretical extensions || to the Standard Model predict D°D mix- 
ing rates high enough to be measurable by existing experiments, making it 
interesting to search for this process. The mixing rate is parametrized as 
r = T(D° — > D° — > f)/T(D° — > f), where / is the final decay state used 
to identify the charm quantum number of the neutral D at the time of de- 
cay. We report here a limit on r using semileptonic decays in the data from 
Fermilab experiment E791. 

Many experiments have used hadronic D° decays to search for mixing. For 
example, Fermilab experiment E691 studied D°D mixing by looking for the 
decay chain D* + —> n + D°, followed by D° — > D° — ► K + n~ or K + n~n + n~ 
[§]. A wrong-sign charged K from the neutral D decay (e.g., D° — > D° —>■ 
K + tt~) can be a signature of mixing. However, a wrong-sign K can also come 
from doubly- Cabibbo-suppressed (DCS) decays in which a D° decays directly 
into the wrong-sign kaon (e.g., D° — > K + tt~). Moreover, the DCS amplitude 
can interfere with the mixing amplitude, reducing the sensitivity to mixing 
[f|] even though the mixing, DCS, and interference terms in principle can be 
separated statistically using decay-time information. E691 reported a 90% 
confidence level (C.L.) limit on r of 0.37% assuming no interference between 
DCS and mixing amplitudes. For worst-case interference, their limit is 1.9% 
in the Ktt mode and 0.7% in the K7nnr mode. E791 has used the same 
decay modes to study D°D mixing using a much larger data sample and 
more general assumptions 0. CLEO II has observed a wrong-sign signal in 
the mode D — > Kir, and measures the ratio of the wrong-sign to right-sign 
decays to be (0.77 ± 0.25 ± 0.25)% ||. However, an unambiguous mixing 
signal cannot be established from the CLEO result because of the lack of 
decay-time information. 

An alternate way to make a mixing measurement is to use semileptonic 
decays to tag the charm of the neutral D at the time of decay. There is no 
DCS amplitude in these decays, eliminating the complications of interference. 
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Fermilab experiment E615 searched for mixing by looking for pairs of muons 
with the same charge in a single event M. Same-sign muons could come from 
the semileptonic decays of a D meson (D° or D + ) and a D° that has oscillated 
into a D°. E615 obtained a 90% C.L. upper limit on r of 0.56% using specific 
assumptions for charm production cross sections and D branching fractions. 

In this letter we report the result of a mixing search using reconstructed 
semileptonic decays of the D° in the data sample of hadropro duct ion exper- 
iment E791 at Fermilab. We observe a large signal for the right-sign (RS) 
decay chain D* + — > tt + D° — > tt + (K~1 + i>i), where I is an e or a //, in which 
the charge of the ir is the same as the charge of the lepton from the neutral 
D decay (charge conjugate modes are implied throughout this paper). We 
search for mixing in wrong-sign (WS) D* + decay candidates in which the 
charge of the ir is opposite to that of the charged lepton. These candidates 
could correspond to the decay chain D* + — > tt + D° — > tt + D° — ► tt + (K + 1~vi). 
We look for two signatures of mixing in the WS sample - a peak in Q-value 
(Q = M(Klun) - M(D°) - M(ic)) at about 5.8 MeV and the characteristic 
distribution in proper decay time t. Assuming mixing is small, the time evo- 
lution of the mixed .D's is given by dN/dt oc t 2 e~ rt , where T is the D° decay 
rate. 

The E791 experiment || recorded 2 x 10 10 events from 500 GeV/c 7r~ in- 
teractions in five thin targets (one platinum, four diamond) separated by gaps 
of 1.34 to 1.39 cm. Precision vertex and tracking information was provided 
by 23 silicon microstrip detectors and 35 drift chamber planes. Momentum 
was measured using two dipole magnets. Two segmented threshold Cerenkov 
counters provided ir/K/p separation in the 6-60 GeV/c momentum range 

1- 

A segmented lead and liquid-scintillator calorimeter is used to identify 
electrons from their energy deposition and transverse shower shape. For 
the cuts used in this analysis, the probability that a tt is misidentified as 
an electron is typically 1.6% and the probability that a, K is misidentified 
as an electron is typically 0.8%. Muon identification is provided by two 
planes of scintillation counters, located behind shower-absorbing calorimeters 
and steel shielding with a total thickness equivalent to 2.5 meters of steel 
(15 proton interaction lengths). All muon candidates are required to have 
momentum greater that 10 GeV/c to reduce background from decays in 
flight, and to leave a signal in the expected scintillation counters, allowing 
for multiple scattering. For the cuts used in this analysis, the probability 
that a 7r is misidentified as a /i is typically 3.6% and the probability that a 
K is misidentified as a /i is typically 4.6%. 

To reduce background, a candidate D° decay vertex is required to be 
separated from the production vertex by at least 8cr z , where a z is the error 
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on the separation between the two vertices. The decay vertex is required 
to be at least 3a away from the edge of the nearest solid material, where 
a is the error on the separation. The minimum parent mass, defined as 



M m i n = pt + \Jvt + where px is the transverse momentum of the Kl 
with respect to the direction of flight of the D° and Mki is the invariant mass 
of the Kl candidates [10J, is required to be in the range 1.6 to 2.1 GeV/c 2 . 
The M min distribution for Monte Carlo signal events has a cusp at the D° 
mass, and falls rapidly at lower values of M min , whereas background rises 
as M min decreases. We also require the invariant mass of the Kl candidate, 
M Kh to be in the range 1.15 to 1.80 GeV/c 2 . The lower cut on M Ki reduces 
noncharm background, and the upper cut on M^i removes feedthrough from 
D° — > Kit decays into the RS sample, in which the ir is misidentified as a 
lepton. We require the transverse momentum of the lepton with respect to 
the direction of flight of the candidate D° to be greater than 0.2 GeV/c and 
that of the hadron to be greater than 0.4 GeV/c, since charm decay products 
tend to have larger such transverse momenta than background tracks. The 
7r + track from the D* + is required to be consistent with belonging to the 
primary vertex and to have momentum greater than 2 GeV/c. 

To eliminate feedthrough of the Kn mode into the wrong-sign sample 
through double misidentification of the hadrons (the K misidentified as a 
lepton and the 7r misidentified as a K), we require \M nK — M D o\ > 30 MeV/c 2 
(typical cr of the D° mass peak in the Kir mode is 15 MeV/c 2 ), where M n K 
is the invariant mass of the Kl candidate when the K is assigned the it mass 
and the / is assigned the K mass. 

Additional cuts are applied to Kfiv candidates because kaons and pi- 
ons are more likely to be misidentified as muons than as electrons due to 
punchthrough and decays in flight. If the muon track is positively iden- 
tified as a kaon in the Cerenkov detectors, the decay vertex is rejected. 
Feedthrough from the mode D° — > K~K + is eliminated by the requirement 
\Mkk — M D o\ > 30 MeV/c 2 , where M^k is the invariant mass of the Kl 
candidate when both tracks are assigned the K mass. We also demand that 
there be one and only one D* candidate (Q- value < 80 MeV/c 2 ) in each event 
in the K\iv sample. An event is rejected if more than one D* candidate is 
found. 

Since there is an undetected neutrino in a semileptonic decay, the D° 
momentum cannot be reconstructed directly. However, using the measured 
positions of the primary and secondary vertices, the measured K and I mo- 
menta, and assuming the parent particle mass is that of a D°, one can solve 
for the neutrino momentum up to a two-fold ambiguity. The solution re- 
sulting in higher D° momentum is used for all events. Monte Carlo (MC) 
studies indicate that it gives a better estimate of the true momentum for the 
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selected sample. From MC, we determine that the root mean square (r.m.s.) 
deviation between the calculated and the true D° momenta is about 15%. 
This also causes smearing in the calculated proper decay time. The effect 
of this smearing is discussed below. Having obtained the D° momentum, 
we calculate the invariant mass of the D* + candidate and the proper decay 
time of the D° candidate. The final Q-value distributions for Kev and Kfiv 
candidates are shown in Figure |l]. 

To search for mixing signals, separate unbinned maximum likelihood fits 
are performed on the Kev and Kfiv samples, using the Q-value and proper 
decay time t for each event. The expected Q- value signal shape in WS 
data is obtained directly from fits to the large, kinematically-identical RS 
signal. It is parametrized by asymmetric Gaussian distributions, broader on 
the high Q side, with widths a decreasing with longer proper decay time t, 
a {t) = J (cr ) 2 + (C/t) 2 . This t-dependence arises because the D° direction 
is measured better for decays at greater distances. The Q-value distribution 
of the background under the D* peak is described by the spectrum which 
results from combining a D° candidate from one event with pions from other 
events to form random D° — it mass combinations (dotted histograms in 
Figure [l]) . 

When smearing is neglected, the measured proper decay-time spectrum 
of a mixing signal is proportional to t 2 e(t)e~ rt , where e(t) is the t-dependent 
detector efficiency. This spectrum is obtained by multiplying the measured 
distribution (crosses in Figure || (a) and (b)) of background-subtracted RS 
data (oc e(t)e _rt ) by the mean value of t 2 in each bin. The t-distribution of 
the non-D* background is obtained from data events in the Q- value sideband 
with 25 < Q < 60 MeV/c 2 . Distributions of t in the three Q- value sidebands 
20 < Q < 30, 40 < Q < 50, and 60 < Q < 70 MeV/c 2 are identical 
within statistical errors. Therefore, the background decay-time distribution 
is constant across the Q-value spectrum, as expected if most background is 
due to real D°'s combined with random pions. Sideband t dependence thus 
can be and is used to model the background in the signal region. 

From the fits, we find N RS = 1237 ± 45 RS events and N mix = 4.4±^J 
WS mixed events in the Kev samples, and Nrs — 1267 ± 44 RS events and 
Nmix = l-8iii]o WS mixed events in the K^v samples. There is no indication 
of a mixing signal. 

The mixing rate is r = (N m i X /Nus) x «, where a accounts for the depen- 
dence of detector acceptance on t and the different t-dependences of mixed 
and unmixed decays. Since vertex reconstruction efficiency is low at small t, 
the detector is more efficient at finding the longer-lived mixed decays. Specif- 
ically, a = (r J °° e(t)e~ rt dt) / (|r 3 / °° e{t)t 2 e~ Tt dt) . It is measured from the 
background-subtracted RS decay-time distribution (oc e(t)e~ rt ) using numer- 
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ical integration. Values of a are 0.44 ±0.02 for the Kev mode and 0.46 ±0.02 
for the K[iv mode. 

We measure the mixing rate to be r = (0.1 61^37)% for the Kev mode 
and r = (0.06^o;4q)% for the Kfxv mode. Taking the weighted average of 
these two statistically-independent results, we get an average mixing rate of 
r = (0.11±8;|?)%. This gives an upper limit for D°D° mixing of r < 0.50% 
at the 90% confidence level. 

Since right- and wrong-sign data samples are selected using identical cri- 
teria, most systematic uncertainties cancel in the mixing rate. Two possibly 
significant sources of systematic error, the time resolution for a mixing signal 
and feedthrough of hadronic decays, remain. 

The decay-time distribution used in the fit for mixed events is propor- 
tional to e(t)t 2 e~ rt , which is valid only if the decay times are measured ex- 
actly. Due to finite resolution of the detector and the choice of one of the two 
neutrino momentum solutions, the measured mixed decay-time distribution 
differs slightly from the distribution used in the fit. The r.m.s. deviation of 
the measured t from the true one is about 15%. MC studies indicate that 
the effect of this smearing in proper decay time on the final result is less than 
10% of the statistical error and hence is ignored. 

Feedthrough of hadronic decays into the semileptonic sample is expected 
to come mainly from modes such as D° — > K~7t + tt° in which an undetected 
neutral hadron approximates the kinematics of a missing neutrino, and a 
hadron is misidentified as a lepton. Misidentified hadronic decays have two 
possible effects: 1) feedthrough into the WS signal would give a false mix- 
ing signal; 2) feedthrough into the RS signal would inflate the denomina- 
tor of the mixing rate, thus overestimating our sensitivity to mixing. WS 
feedthrough requires K-n misidentification as well as hadron-lepton misiden- 
tification. Another source of feedthrough into the WS sample is doubly 
misidentified semileptonic decays (in which the hadron is misidentified as 
a lepton and the lepton is misidentified as a kaon). We see no WS signal 
and make no correction for this effect. RS feedthrough has been modeled by 
Monte Carlo simulation, and shown to be about 3% of the RS signal. No 
correction is made for this effect. 

To check our fitting procedure and to look for possible systematic effects 
in our sensitivity, we added fixed numbers M (M = 10, 20, 30, 40; the typical 
fit error on our N mix is 10) of simulated mixed events to the wrong-sign data 
sample and refit. These fits found a mixing rate systematically (10 - 15)% 
higher than the correct value, mainly due to an overestimate of the correction 
factor a which is a result of our choice of the neutrino momentum resulting 
in higher D° momentum. We conservatively choose not to correct for this 
systematic overestimate of mixing rate. 
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In summary, we have searched for D°D° mixing using D* + — > ti + D — > 
7T + .D — > 7i + (K + l~vi) candidates together with decay time information. We 
obtain a 90% CL. upper limit of 0.50% on the mixing rate. This result is 
the best mo del- independent limit on D°D mixing to date. 
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U.S. Department of Energy, the U.S. -Israel Binational Science Foundation, 
and the U.S. National Science Foundation. Fermilab is operated by the Uni- 
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Department of Energy. 
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Figure 1: The Q- value distributions for (a) Kev RS, (b) K/iu RS, (c) Kev 
WS, and (d) Kfxv WS candidates. The solid line histograms show the data Q- 
value distributions, the dashed lines are the projections of the fit in Q- value, 
and the dotted lines show the Q-value distribution obtained from combining 
a D° from one event and n from another, normalized to the number of events 
with Q > 0.025 GeV/c 2 in the respective histograms. 
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Figure 2: Dependence of RS and WS signals on proper decay time t. Crosses 
represent the measured decay-time distributions for (a) Kev and (b) K/iu 
background-subtracted RS signals, and for (c) Kev and (d) K/iv WS signal 
region (Q < 0.015 GeV/c 2 ). The dashed line in all histograms is the expected 
D° decay-time distribution uncorrected for detector acceptance, normalized 
to the number of events with t > 0.7 ps (where acceptance is uniform). The 
dotted lines in (c) and (d) represent the expected decay-time distributions 
uncorrected for detector acceptance for Kev and K/iv mixing signals, corre- 
sponding to our 90% C.L. limit in each mode. 
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